It is known to use a variety of devices to display images to a user. As used herein, an image may be a field (e.g., such as in the context of an interlaced image) or a frame (such as in the context of a deinterlaced or progressive image). Televisions, media players (e.g., DVD players), computing devices (e.g., stationary desktop computers), mobile devices (e.g., cellular phones, personal digital assistants, etc.) are each capable of displaying sequences of images (video and/or graphics) to a user. In the context of video, image sequences are generally created using a fixed image rate (a.k.a. frame or field rate, when dealing with frames or fields, specifically). For example, many telecine processes record video (e.g., movies) in real time, i.e., having a image (or frame) rate of 24 Hz. When the video is intended to be displayed on a display device that has a refresh rate that does not match and is faster than the image rate of the source video, the display device must create new images for insertion into the destination (i.e., the output) video stream. For example, many standard television sets in North America refresh at 30 frames per second while many standard computer monitors (e.g., an LCD display) refresh at 60 Hz (and may be as high as 120 Hz).
When a movie having a 24 Hz frame rate is displayed on a display having a refresh rate of 60 Hz, the source sequence of images must be “upconverted” to 60 Hz. The most common method is to simply repeat a previously known frame at a predetermined point in time. This is often accomplished using a 3:2 pulldown where a first image “2n” is repeated three times and a second image “2n+1” is repeated 2 times in a sample and hold process. FIG. 1 illustrates an example of a sample and hold process for a generic upconversion (not specifically an upconversion from 24 Hz to 60 Hz). As shown, the top row represents a source sequence of images including images A, B and C and has a first image rate such that image A is displayed at time TS1, image B is displayed at TS2 and image C is displayed at TS3. The lower “row” of images displays the sequence of upsampled images for a display that has a higher refresh rate or image rate. Thus, upconversion is required for display. The destination sequence of images has a second image rate that is higher or faster than the first image rate such that in this generic example, the first image A is displayed on the display at time TO1 and is repeated at TO2. Subsequently, image B is displayed at time TO3 and again at TO4. Image C is displayed at time TO5, etc.
As is known, sample and hold upconversions such as the 3:2 pulldown create annoying artifacts called motion judder effects in areas where objects move with respect to consecutive images in the source image sequence. In other words, the motion of objects in consecutive images is not smooth in prior art sample and hold upconversions. As a result, the human eye has difficulty focusing on it.
To mitigate against these annoying judder effects, a variety of techniques have been established. The first group of techniques use non-motion compensation such as linear interpolation between two original images. These techniques, however, fail because motion judder is still perceived by the human eye. The second group of techniques are motion compensation based. These techniques attempt to display a moving object in the correct spatial position in the upconverted new image. For example, FIG. 2 shows this concept where an object in a first image A is initially displayed in a first location 202 moves to a new location 204 in second image B, where frames A and B are from the source sequence of images. The upconverted new image Z is designed to place the object in the correct location 206 along the motion trajectory 208.
A first type of motion compensation upconversion is the block matching method that is akin to motion estimation in MPEG, a type of video compression algorithm. Accordingly, block matching techniques are often affiliated with video compression engines. The block matching method takes a first image and a second image of a source sequence and looks for similar blocks within the images. The detection of similar blocks indicates that motion of pixels associated with these blocks. The block matching method, however: (1) is not accurate at the sub-pixel level without using expensive processing resources; (2) is expensive to avoid error; and (3) requires recursive algorithms. As is recognized, if an object moves a non-integer number of pixels (e.g., 2.5 pixel motion), the block matching method requires a pair of upscalers to convert, to a certain degree, subpixel resolution to pixel resolution and then performs the block matching to obtain the amount of motion (in integer pixels) in the upscaled pixel resolution, which corresponds to the subpixel resolution (with fractional pixel) in the original un-upscaled image. Additionally, in order to avoid any error in detection of motion, expensive and recursive processing is required to ensure that the allegedly similar blocks in the second frame were moved from the first frame. In other words, the block matching methods require expensive and recursive processing algorithms to properly “match” the moved blocks. Because block matching methods are not always accurate at the sub-pixel level and because they require expensive and recursive algorithms to implement to avoid error, block matching methods are not desirable.
Another type of motion compensation upconversion is the phase plane correlation method that detections motion in the phase-domain and therefore requires fourier transforms on the images and inverse fourier transforms (e.g., using FFTs and iFFTs) to properly identify and verify motion in images. Because implementation of fourier transforms is computationally expensive, phase plane correlation methods are not desirable.
Yet another type of motion compensation is the optical flow method that measures motion vectors over two consecutive fields using gradient-based and recursive algorithms (in the x, y and t dimensions) to optimize the pixel motion vectors. This method for upconversion is expensive to implement due to its recursive nature. Consequently, the optical flow method of the prior art is also not desirable.
Therefore a need exists to upconvert a source sequence of images to create a destination or output sequence of images having a higher image rate than the source sequence of images while avoiding the expensive collateral consequences of prior art methods.